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ABSTRACT 


Acquisition costs of Department of Defense (DoD) weapon systems are of major 
concern due to their enormous annual expense. In fact, $165B was spent by 
DoD on acquisition in 2008. However, another budget component, Product 
Support, costs the government an additional $132B annually. DoD has stated 
that the preferred method for providing this life-cycle sustainment is Performance 
Based Logistics (PBL), and 20% of all programs now use that method of support. 
PBL’s goal is to provide maximum readiness at reasonable costs. 


Unfortunately, evaluation of the success of these PBL contracts is difficult, 
because of the large number of metrics and vast differences among weapon 
systems. Several studies have attempted to do so, but they have not used DoD- 
prescribed metrics for evaluation. This research attempts to use two of the five 
DoD-prescribed metrics to analyze the PBL contract for the Army High Mobility 
Medium Wheeled Vehicle (HMMWV) Integrated Logistics Plan (ILP). These two 
are Cost Per Unit Usage (CPUU) and Operational Availability. 


This statistical analysis found that the ILP does not appear to be 
decreasing cost or increasing Operational Availability for the Army HMMWV 


program. Further study using the other three DoD metrics is recommended. 
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EXECUTIVE SUMMARY 


Acquisition costs of DoD weapon systems are of major concern due to their 
enormous annual expense. In fact, $165B was spent by Department of Defense 
(DoD) on acquisition in 2008. However, another budget component, Product 
Support, costs the government an additional $132B annually. DoD has stated 
that the preferred method for providing this life-cycle sustainment is Performance 
Based Logistics (PBL), and 20% of all programs now use that method of support. 
PBL’s goal is to provide maximum readiness at reasonable costs. 


In October 2009, President Obama signed the Fiscal Year 2010 National 
Defense Authorization Act. This act contained a provision entitled "Life-cycle 
Management and Product Support" requiring that: (1) the Secretary of Defense 
issue comprehensive guidance on Life-cycle Management (LCM), and on the 
development and implementation of product support strategies for major weapon 
systems; (2) each major weapons system be supported by a Product Support 
Manager (PSM); and (3) each PSM position be performed by a properly qualified 
member of the armed forces or full-time employee of the DoD. Product support 
and sustainment thus became a high priority for DoD. 


Since 1999, the DoD has been transforming to a system that places more 
emphasis on readiness outcomes. Performance-Based Logistics (PBL) has been 
named the preferred method of life-cycle sustainment. PBL has allowed 
partnerships between government and industry to be formed. These 
partnerships are designed to bring together the best of the public and private 


sectors in order to provide DoD with maximum readiness at reasonable costs. 


The Army has been working through this transformation in life-cycle 
sustainment. Specifically in 2006, it implemented an Integrated Logistics Plan for 
the recapitalization (RECAP) program for High Mobility Medium Wheeled Vehicle 
(HMMWY) vehicles. This created a partnership with AM General to provide parts 
to Army Depots conducting the RECAP of HMMWVs. This study analyzes the 


XV 


Cost Per Unit Usage (CPUU) and Operational Availability (Ao) of the Army 
HMMWV. The CPUU and Operational Availability were analyzed over a six year 
period from 2004 to 2009. 


The primary goal of this thesis was to conduct a comparison of different 
support strategies on a weapon system. Data was collected from the Army 
Logistics Integrated Database (LIDB) and Operating and Support Management 
Information System (OSMIS). This study focused on the M998 and M1097 
models, which are two of the models that went through the RECAP process. The 
data for the usage (mileage) of the HMMWV’s contained numerous anomalies 
and was missing numerous odometer readings. This made computing the CPUU 
difficult, as we had to make several corrections to the usage data under 


assumptions that we determined to be reasonable based on the usable data. 


Ultimately, we found that the ILP does not appear to be providing any 
costs savings or increased Operational Availability since its implementation. The 
CPUU had an upward trend over the six-year period and the pre-ILP cost was 
found to be cheaper than the post-ILP cost. The Operational Availability was 
consistent at 90% to 95% across this same time period. The major finding in this 
study was that the data available to conduct these type of studies is poorly 
maintained. The amount of missing and anomalous data does not allow analysts 
to confidently draw conclusions. It is recommended that DoD invest time and 
money in data collection and database organization in order to provide analysts 


with accurate and complete data. 


Studies such as this can provide valuable information to key decision 
makers within the DoD. However, in order to provide that valuable information, 
analysts need to be provided with complete and accurate data. Providing our 
warfighters with the best weapons systems at reasonable costs to the taxpayer is 
a key issue in the United States. It is critical that DoD analysts be provided with 
cost and availability information to allow them to make accurate estimates of 


weapons system support. 
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I. INTRODUCTION 


Acquisition costs of DoD weapon systems are of major concern due to 
their enormous annual expense. In fact, $165B was spent by DoD on acquisition 
in 2008. However, another budget component, Product Support, costs the 
government an additional $132B annually. (U.S. Department of Defense, Deputy 
Under Secretary of Defense for Logistics and Materiel Readiness, 2009, p. 6) 
DoD has stated that the preferred method for providing this life-cycle sustainment 
is Performance Based Logistics (PBL), and 20% of all programs now use that 
method of support. PBL’s goal is to provide maximum readiness at reasonable 


costs. 


For years product managers managed components of readiness as 
discrete units. In other words, they bought and stocked parts, systems, and sub- 
systems, and shipped them to units as necessary. Since 1999, the DoD has 
been evolving to a system that places more emphasis on readiness outcomes. 
Performance-Based Logistics (PBL) is the “poster child” of this strategy. PBL 
has allowed partnerships between government and industry to be formed. These 
partnerships were designed to bring together the best of the public and private 
sectors in order to provide DoD with maximum readiness at reasonable costs. 
While this strategy has been declared as the preferred means of life-cycle 
support (U.S. Department of Defense, 2007, p. 7), fewer than 20% of all 
weapons systems are using this strategy. PBL has been criticized as simply 
being “contracted logistics.” Many have questioned the cost-effectiveness of this 
strategy. The Government Accountability Office (GAO) has conducted two 
separate studies on PBL and has yet to find any conclusive evidence that PBL 
contracts are saving the government money. Other studies have indicated 
increased levels of availability. Fewer people have raised questions concerning 
this metric than with cost. Lawmakers, DoD administrators, and our Military 


Commanders need to be able to look at the costs and availability together to 
determine if these types of support strategies are the future of weapon systems 


life-cycle sustainment. 


A. BACKGROUND 


In 2005, an Integrated Logistics Plan (ILP) was developed in an attempt to 
increase readiness for the Army HMMWV. Two production lines, Letterkenny 
Army Depot (LEAD) in Franklin County, Pennsylvania, and Red River Army 
Depot (RRAD) in Bowie County, Texas, were chosen for the program. This was 
expanded to include Maine Military Authority (MMA) in Aroostook County, Maine, 
in May of 2006. (Smith, 2009, p. 6) The idea behind the ILP was to move the 
“point of sale” to the “point of use” by having the supplier deliver the parts and 
having minimal government intervention in the supply chain. AM General was 
awarded a $200 million contract to provide parts for the HMMWV recapitalization 
program at the two Army Depots. 


The overall strategy of the program was to optimize support to the 
warfighter, which is also the overall goal of the DoD policy on weapons system 
support. Additionally, this partnership was expected to save the Army money on 
Operations and Support (O&S) costs. The contract with AM General was 
renewed in 2008. The Army has reported an overall savings of $4,520 and 
$3,414 per vehicle at LEAD and RRAD, respectively. Considering the large 
number of HMMWV’s the Army has in its inventory, this is a substantial savings 
to the DoD. Army Material Systems Analysis Activity (AMSAA) reported an 
overall savings of over $800,000 for FY06. (Chapman, 2007) 


The Army and other services continue to modify their maintenance 
strategies to try and find ways to increase availability and save costs. Each of 
these strategies needs to be analyzed to determine effects on costs and 
availability and to give decision makers a basis for their future plans. 


B. ANALYSIS 


The key metrics of this study were Cost Per Unit Usage and Operational 
Availability. The years 2004-2005 were used as a baseline, since these were 
the two years of data prior to the ILP being put in place. Then the years 2006-— 
2009 were compared to the baseline to determine if there were any costs savings 
or increased Operational Availability. We also looked at the trends across the 
entire time frame to determine what the CPUU and Operational Availability were 
doing over time. All the analysis was conducted in R (R Development Core 
Team, 2010) and Microsoft Excel (Excel, 2007). 


The largest portion of our time was spent handling the data. In particular, 
the usage data required substantial repairs due to missing and anomalous data. 
The serial numbers for the vehicles also required several repairs mostly due to 
obvious entry errors. Once the data was corrected and put in a usable format, 
we used several different strategies to examine the effects of the ILP. For 
instance, the metrics were categorized by Major Army Command (MACOM) to 
eliminate the possibility that some of the smaller commands had too much 
leverage on the data. We used statistical tests to determine if the differences 
between the time periods were statistically significant. The data was broken 
down by quarters and analyzed to determine trends over the course of the study 


period. Ultimately, all the analyses led to similar conclusions. 


C. SUMMARY OF FINDINGS 


The Cost Per Unit Usage (CPUU) was found to have a rising trend over 
the time period studied. When comparing the two time periods, pre-ILP and 
post-ILP, the study determined that the pre-ILP CPUU was less expensive. 
Several different analyses were conducted and all of them came to the same two 
conclusions: the Army HMMWV ILP is not saving the DoD money, and the 


Operational Availability has not increased. 


The Operational Availability was studied in a manner similar to that of the 
CPUU. We found that over the period studied, the Operational Availability 
3 


showed a very slight downward trend. However, the decrease was insignificant 
and Operational Availability never dropped below 90% during the time-period 
studied. 


In conducting these analyses, the major finding was that the data was 
inconsistent and missing numerous values. Specifically, the usage data, which 
was vitally important to studying the CPUU, was missing monthly entries and had 
numerous obvious errors, such as a monthly reading being lower than the 
previous monthly reading. Problems like this caused difficulty and lots of data 
had to be eliminated or corrected in order to conduct the analysis. 


D. ORGANIZATION OF THIS THESIS 


This thesis describes the analytical methodology, findings, and 
recommendations for achieving long-term product support for the Army HMMWV. 
Chapter II of this thesis provides a brief synopsis of the different reports and 
analyses that were read while conducting this study. Chapter III provides an in- 
depth discussion of the different data sources used and, more importantly, the 
problems encountered with the data. The analysis approach, methodology, and 
resulting models are present in Chapter IV. Finally, Chapter V closes the report 
by drawing conclusions and making recommendations for a successful future of 


the HMMWV and other DoD weapons systems. 


ll. LITERATURE REVIEW 


Since the inception of Performance-Based Logistics, a number of studies 
have examined its performance. The Government Accountability Office (GAO) 
has conducted two separate analyses of PBL’s, one in 2005 and one in 2008. 
(U.S. Government Accountability Office, 2005, 2008) The RAND Corporation 
has also done studies on PBL, as have many other analysis organizations, 
including the U.S. Army Materiel Systems Analysis Activity, the Office of the 
Under Secretary of Defense, independent industry, and academia. (RAND 
Corporation, 2008; Chapman, 2007; U.S. Department of Defense, Deputy Under 
Secretary of Defense for Logistics and Materiel Readiness, 2009). But few of 
these studies have examined the DoD-prescribed metrics for their analysis. 


A. PRODUCT SUPPORT ASSESSMENT TEAM 


The most recent high-level study conducted on Performance-Based 
Logistics was released by the Under Secretary of Defense for Acquisition, 
Technology and Logistics (USD AT&L) in November of 2009. This report was 
created by a working group of military, civilian DoD, industry, and academic 
analysts designated as the Product Support Assessment Team (PSAT). While 
the report was very broad and covered more topics than just costs and 
availability, one of the major findings was: “Performance-based product support 
strategies consistently deliver improved materiel readiness, but assessing the 
true cost of both traditional and performance-based strategies is difficult, if not 
impossible, given current financial systems.” (U.S. Department of Defense, 
Deputy Under Secretary of Defense for Logistics and Materiel Readiness, 2009, 
p. 9) The major issue the PSAT had with this analysis was that the data for the 
cost analysis was either unavailable or inconclusive. Furthermore, across the 


services, there was no standard way of tracking costs. 


Another interesting finding from this report was the fact that acquisition 


and sustainment communities do not have shared goals and are funded 
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separately. (U.S. Department of Defense, Deputy Under Secretary of Defense 
for Logistics and Materiel Readiness, 2009, p. 26) For example, acquisition 
organizations have traditionally purchased weapons systems from the lowest 
bidder. While this may provide savings for the procurement portion of the life 
span of the system, it may, in turn, incur more Operation and Support (O&S) 
costs later in the life-cycle. If the acquisition groups and logistical support groups 
worked more cohesively and had common goals, life-cycle cost savings could be 
more easily achieved. In many cases, more costs up front could lead to large 
savings in O&S costs over the life span of the system. PBL contracts force the 
acquisition and logistic organizations to work together and come to agreement on 


some common goals. 


The PSAT was asked to conduct the analysis on analogous systems. In 
other words, they grouped systems by mission capability, such as land assault 
(AAV, Bradley, Stryker, M1 FOV, and M1 Abrams), land transport (FMTV, MTVR, 
HEMTT, HMMWV (Army), and HMMWV (USMC)), etc. However, comparing 
different systems, produced by different companies and at different stages in 
their life cycles, is problematic. Ideally, a single system could be examined both 
with PBL in place and without. For example, a lower cost per unit usage for the 
HMMWV than for the MTVR might suggest that the former’s maintenance plan is 
more efficient. However, the MTVR might be inherently more expensive to 
maintain, perhaps because parts cost more or because the vehicles are used 
differently. The PSAT used two primary indexes for grading the systems: 
Sustained Readiness Improvement (SRI) and Sustained Cost Management 
(SCM). The SRI is a binary metric that was calculated by taking the average 
availability change of all systems within a category as the baseline. An 
individual system earned a value of one for any year in two distinct ways. First, a 
system scored a one if it experienced an increase in availability from one year to 
the next. Second, a one was also scored if the system decreased in availability 
at a rate less than the average availability change for that category. Otherwise, a 
zero was scored for that year. The average of the ones and zeros over the 


6 


period study produced the SRI. (U.S. Department of Defense, Deputy Under 
Secretary of Defense for Logistics and Materiel Readiness, 2009, p. 83) The 
SCM was calculated using a similar method except that Cost Per Unit Usage 
(CPUU) was used for each system and in order to earn a one, the system’s cost 
had to decrease from year to year or increase at a rate smaller than the average 
CPUU change. An example of why this could be a problem is shown in Table 1. 


The PSAT found numerous flaws within DoD that needed to be corrected 
and made several suggestions on how improving this process could be 
implemented. It also pointed out that this would require strong leadership, open- 
mindedness, a reform-driven DoD-Congressional partnership, and a collaborative 
DoD-Industry relationship. (U.S. Department of Defense, Deputy Under 


Secretary of Defense for Logistics and Materiel Readiness, 2009, p. 12) 


Table 1 shows how the SRI used by PSAT could give a false indication 
that two systems are performing at a similar level. Consider two systems with 
Availability, as shown in Table 1. 


System 1: 
Availability 
% Change 


Score 


System 2: 
Availability 
% Change 


Score 


Average 


% Change 


Year 1 Year 3 Year7 Average 
80% 78% 78% 78% 
—1% +1% 
1 1 


Year 1 Year 3 Year7 Average 
74% 60% 65% 67% 
—20% +2% 
0 1 


77% 77% 69% 71% 
0% —11% +3% 


Table 1: SRI for Two Different Weapons Systems 


Readiness improvement 


+ 


4 


Readiness improvement 
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So, according to the PSAT results, both systems receive a grade of 67%; 
however, clearly system #1, which has an overall average of 78% availability, is 
the better system. A graph of the corresponding Sustained Readiness 


Improvement and the Operational Availability are shown in Figure 1 and Figure 2. 
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Figure 1: Sustained Readiness Improvement versus Time for Two 
Weapons Systems and Their Baseline Index 
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Figure 2: Operational Availability for Two Weapons Systems Over Time 


B. GOVERNMENT ACCOUNTABILITY OFFICER 


The GAO has released two reports regarding Performance-Based 
Logistics. (U.S. Government Accountability Office, 2005, 2008) These reports 
reach similar conclusions. The GAO found that while DoD guidance states that 
program offices should use business case analysis (BCA) to guide decision 
making regarding PBL contracts, the services are not consistent and in many 
cases simply did not conduct a BCA. 


In the 2005 study, only one of the fifteen program offices reviewed actually 
updated the BCA in accordance with DoD instruction. One of the two major 
recommendations in this report to the DoD was “to reaffirm DoD guidance that 
program offices update the business case analysis following implementation of a 
performance-based logistics arrangement and develop procedures, in 
conjunction with the military services, to track where program offices that enter 
into those arrangements validate their business case decisions consistent with 
DoD guidance.” (U.S. Government Accountability Office, 2005, p. 12) Three 
years later, of the 29 PBL arrangements the GAO studied, about half of them 
either did not use a business case analysis or were unable to provide 
documentation of one. (U.S. Government Accountability Office, 2008, p. 5) 
Clearly, the problem with business case analysis has not been fixed. It may be 
that the guidance for these business case analyses needs to be revised or 
perhaps the program offices are not appropriately equipped to conduct them. 
This is an area that still needs to be addressed and corrected in order for studies 
such as this to make any substantial conclusions regarding the effectiveness of 
PBL contracts. 


The GAO also found in both reports that the data used to evaluate these 
contracts is not sufficient to draw conclusions. A common belief identified in 
these reports was that a fixed cost contract would always be cheaper than a 
traditional contract and therefore the program offices did not bother tracking 


costs. The program offices also acknowledged limitations in their own 
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information systems in providing reliable data to monitor contractor cost and 
performance. Furthermore, they stated that they had more confidence in the 
contractors’ systems than in their own regarding accuracy and completeness. 
(U.S. Government Accountability Office, 2005, pp. 9-10) According to the 2008 
report, this same problem persisted. It stated, “Program managers often lacked 
detailed and standardized cost data.” (U.S. Government Accountability Office, 
2008, p. 32) We encountered this same lack of data problem as well, and an 
entire chapter of this study has been dedicated to the different problems 
encountered. Ultimately, the DoD has to lay down and enforce the standards 


established in order to track this data accurately and completely. 
C. RAND CORPORATION 


In 2008, the RAND Corporation released a study conducted on the Army 
HMMWV recapitalization (RECAP) program. As with the current study, the 
RAND study was performed at the vehicle level. The key objectives of the study 
were to assess the effects of age on costs and availability, develop a tool that 
determines optimal RECAP times, and demonstrate how the tool can be used to 
produce recommendations. (RAND Corporation, 2008) While RAND’s analysts 
were not as interested in cost per unit usage as this study is, they did determine 
that the optimal replacement point for the M998 occurred at age 12, yielding an 
average cost per mile of $5.53 over the lifetime of the vehicle. (RAND 


Corporation, 2008, p. xiii) 


As with all the other studies, RAND found numerous problems with the 
data. Since their study was based on age, the Year of Manufacture (YOM) was 
important data. About 5% of the data they collected did not have a YOM, or it 
contained a YOM that was obviously incorrect. The usage data was even worse. 
In order to put some realistic maximum limitations on this, they capped the usage 
at 3,000 miles per month. With this restriction, over 24% of the data had missing 
or invalid usage elements. RAND used simple imputation to replace these 


missing values. Another issue RAND faced lay in the part order data. Their data 
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contained part orders that were not true HMMWV parts, and therefore those were 
excluded. (RAND Corporation, 2008, p. 57) Fortunately, RAND’s data set was 
large enough that even with these problems, they were able to conduct a 


thorough study and develop a model for determining the optimal age for RECAP. 
D. ARMY MATERIEL SYSTEMS ANALYSIS ACTIVITY (AMSAA) 


The Army Materiel Systems Analysis Activity (AMSAA) conducted a study 
on the HMMWV ILP and released a report in October of 2007. This study was 
conducted for Headquarters Army Materiel Command and included five different 
performance metrics. These performance metrics were: Stockout Rate, 
Turnaround Time, Code G Vehicle Rate, Customer Wait Time, and Quality 
Defect Rate. There were also four cost metrics: Repair Parts Cost, Supply Chain 
Operating Cost, Investment/Other Additional Cost, and Inventory Holding Cost. 
According to this study, the cost savings to the government from the inception of 
the ILP in January of 2006 to December of 2006 was approximately $800,000. 
(Chapman, 2007, p. 33). The performance of the program conveyed consistent 
improvement according to AMSAA. The major recommendations that AMSAA 
had upon conclusion of their study all centered around data collection. They 
found that the data for the baseline assessment was not available, that adequate 
time to conduct the baseline data collection was not permitted prior to the 
performance based agreement (PBA) being negotiated, and that baseline metrics 
needed to be established prior to the PBA being written. As with all other studies 
that were reviewed, the lack of data or inaccuracy of data was a major hurdle to 


overcome. 
E. DEFENSE ACQUISITION UNIVERSITY 


In order to gain insight into Performance-Based Logistics, the “PBL Guide” 
provided by Defense Acquisition University was reviewed along with the 
governing DoD instruction, 5000 series. According to DoD Instruction 5000.01, 
program managers “shall develop and implement performance-based logistics 


strategies that optimize total system availability while minimizing costs and 
12 


logistics footprint.” This is interesting to note, considering that, as previously 
stated, only 20% of current weapons systems are under a PBL contract. 
According to the “PBL Guide” and many other reports, the preferred method of 
product support strategy is performance-based logistics. (Defense Acquisition 
University, 2005, p. vii) 

The “PBL Guide” offers information for program managers to use as they 
shift to this new strategy. It offers guidance on developing a team, determining 
performance outcomes, supply chain management strategy, and awarding 
contracts, along with many other topics. Additionally, this guide provides 
information on the five primary metrics for PBL contracts: 1) Operational 
Availability, 2) Operational Reliability, 3) Cost per Unit Usage, 4) Logistics 
Footprint, and 5) Logistics Response Time. Also presented in this guide book 
are several success stories of PBL contracts and a section on best practices 


determined from these success stories. 
F. CONCLUSION 


In conclusion, many different studies were reviewed and one resounding 
conclusion consistently appeared. The data that was available to conduct cost 
and performance analysis is limited and of dubious quality. Because of this, the 
ability of the different teams to provide substantial and conclusive results was 
very limited. Furthermore, they were unable to study the DoD prescribed metrics 
for PBL contracts. In the case of the PSAT study, the scope of the study was 
extremely large and made an attempt to compare analogous systems. 
Unfortunately, while these systems are similar, they have their differences as 
well, and therefore it is hard to make exact comparisons. Ultimately, to 
determine if PBL contracts are working, analysts will have to study systems that 
have used both methods of life-cycle support. This allows for a fair comparison 
amongst the two methods. Also, the metrics prescribed by DoD should be the 
focus of these future studies. The bottom line is that in order to conduct a 
thorough and significant analysis, there must be accurate and complete data 
available. 
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lll. DATA ISSUES 


A major difficulty in this study in assessing the metrics for the HMMWV is 
the inconsistencies found in the data. There are many obvious errors that were 
found within the data sets. This problem was discussed in Chapter Il. The data 
collection process requires drastic improvement. Studies such as this are only 
as good as the data they can use. This chapter discusses the problems in the 
data that was obtained on the HMMWV for both the U.S. Army and U.S. Marine 
Corps. 


A. UNITED STATES ARMY 


The metrics studied for the HMMWV were Operational Availability and 
Cost Per Unit Usage (CPUU). These metrics were explored on the M998 and 
M1097 models, which are the two models for which the Army ILP was originally 
written. The data was collected from Operating and Support Management 
Information System (OSMIS) and the Logistics Integrated Database (LIDB). The 
data collected from OSMIS provided a summary of quarterly costs for each Major 
Army Command (MACOM) and we were able to capture data from 1993 to 
present. LIDB provided a detailed look at costs broken down by individual Work 
Order Numbers (WONs). LIDB also provided usage data for each vehicle by 
serial number. The Operational Availability data in LIDB is reported by each Unit 
Identification Code (UIC) that maintains that particular style of vehicle and is 
recorded monthly. All data collected from LIDB included the years 2000 to 2009. 
However, due to minimal data recorded between 2000 to 2003, only 2004 to 
2009 were analyzed. 


1. Operating and Support Management Information System 
(OSMIS) 


The data collected from OSMIS had numerous anomalies that had to be 
studied closely to determine why they occurred or whether the data was simply 


incorrect. For instance, 10% of the line items in the OSMIS data contained zeros 
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in the Activity (amount of usage) field. This typically would cause the Cost Per 
Unit Usage, which is the sum of all the costs for a vehicle divided by the miles 
traveled, to be infinite. However, these values were recorded as zero. This 
clearly is an error because there were reported costs for those years and 
therefore the vehicles had presumably been used. It should be noted that most 
of these zeros fell in the time frame from 1993-1998. Furthermore, the 
maximum CPUU during the time frame studied was $16.4 billion. Approximately 
10% of the reported CPUU values were over $10.00 per mile. Such a vehicle 
operated for 1200 miles per year would cost around $12,000 dollars per year; 
over the course of a 15-year life span, O&S costs would be around $180,000 
dollars at that rate. That does not include the purchase price. From these 
observations of the CPUU reported in OSMIS, we decided that this 10% of the 
data was defective and therefore needed to be removed. A plot of average 
CPUU for Army HMMWV is shown in Figure 3. 
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Figure 3: Army HMMWV CPUU for M998 and M1097 from 1996 to 2000 
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The CPUU for the M998 is much more consistent than that of the M1097 
and this can probably be attributed to the fact that there are substantially more 
M998 vehicles than M1097 vehicles for most of the years studied. One last 
interesting note for the CPUU is the drastic spike in 2009 for the M998 model and 
in 1999 and 2001 for the M1097 model. 


Another example of the inconsistencies lay in the number of HMMWV’s 
reported in OSMIS that are displayed in Figure 4. This type of study requires a 
large number of vehicles in order to ensure accurate results. In the final data 
sets used for the study, there were ample vehicles, but the point of this 
paragraph is to show inconsistencies in the data. Note, the drastic change in the 
reported number of M998 models between Quarter 4 of 1998 and Quarter 1 of 
1999. This drop was over 30,000 vehicles. While there is no clear explanation 
for this, it could possibly be attributed to some vehicles being re-designated to a 
different model number due to an upgrade or addition to the vehicle. Numerous 
calls and emails yielded no other possible conclusions from the Program 
Manager and AMC. For example, if a new communication system is installed on 
a vehicle, it is re-designated from M1097 to M1097A1. Also notable in this graph 
is the spike in numbers for the M1097 during 1998 followed by a drop in 1999. 
While the drop is not nearly significant as that for M998 in 1999, there is another 
substantial drop between Quarters 1 and 2 in 2006 for the M998 model. This is 
when the ILP first began. One trend that is clear from this graph is the rise in 
numbers of M1097 models and decrease of M998 models beginning in 2006. 
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Number of M1097 


The final area of interest from the OSMIS data was the Cost Per 
System/Unit (CPU). CPU is the total O&S costs for an individual system. While 
this was not one of the primary metrics studied, the data was observed initially 
just to see how it coincided with the Cost Per Unit Usage. It seems that these 
two metrics should have some similarities especially over such a large sample 
size. The graph of the CPU is displayed in Figure 5. While there appears to be a 
rising trend across the entire time frame, it is also easy to see the large spikes 
from quarter to quarter. Furthermore the increases and decreases do not 


coincide with the increases and decreases in the CPUU graph in Figure 3. 
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Army HMMWV Cost Per Unit 
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Figure5: Army HMMWV Cost Per Unit for M998 and M1097 from 1996 to 2009 
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While there appears to be no consistency in this data, the two models do 
seem to follow similar trends. In other words, CPU for the M998 and M1097 
models appear to be rising and falling at the same time. From 2000 to 2007, the 
range of CPU for M998 was $779.52 to $1528.49. This is a significant range, 
particularly considering that all these values have been adjusted to FY 2011 
dollars. 


2. Logistics Integrated Database (LIDB) 


The Logistics Integrated Database contained vehicle level data that was 
much more useful. Data on both cost and Operational Availability was obtained 
for the years 2000-2009. Although this was by far the best data we were able to 
obtain, it still had some significant problems. First, the cost data for these years 
had 268,348 Work Order Numbers with 294 of those being duplicates. However, 
the categorical data had only 264,492 WONs with 59 duplicates. Costs 
associated with a WON that did not exist in the categorical data raised some 
concern. Without a WON in the categorical data, it was impossible to determine 
which serial number those costs could be associated with. 


The next major problem with this data was the fact that the first four years 
of the data contained very few WONs. That left us with only two-years’ worth of 
data to establish the baseline before the ILP was implemented. A histogram of 
the number of WONs by year is shown in Figure 6. In 2000, there were two 
WONs in the data; in 2001 only one; and there were sixteen in 2002. These 
numbers are not visible in the graph due to the scale. The important conclusion 
is that prior to 2004, WONs were not being tracked or entered, and therefore, 


these years cannot be used for any significant analysis. 
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Histogram of WONs by Year 
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Figure 6: Histogram of Work Order Numbers for 2000-2009 


Finally, of the over 250,000 work orders associated with costs, over 
12,000 of them had a zero in the total work order cost field, and over 26,000 of 
them were missing the serial number for the vehicle. Slightly fewer than 140,000 
of them had valid six-digit serial numbers (before we implemented correction 
algorithms). Ultimately, this was the best we were able to collect and after 
corrections to serial numbers, that is detailed in Appendix A, we used just under 
180,000 WONs for this study. 
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B. UNITED STATES MARINE CORPS 


1. Visibility and Management of Operating and Support Costs 
(VAMOSC) 


The metrics studied for the USMC HMMWV were the same as with the 
Army: Operational Availability and Cost Per Unit Usage. The Marine Corps data 
was gathered from the Navy Visibility And Management of Operating and 
Support Cost (VAMOSC) database at the Naval Center for Cost Analysis 
(NCCA). Data was gathered on eight different models of the HMMWV. The 
years from 1998 to 2008 were available for download; however, some of the 
models did not appear in all of these years. The data obtained was summary 
statistics by model and year. There were 88 different combinations of model and 
year obtained. As with the Army data, several anomalies were noted upon initial 


collection of the data. 


The most significant problem with this data set was the fact that the 
“OPTEMPO” or usage was reported for both deployed and regular units; 
however, the inventory was only reported for the entire fleet of that particular 
model. This makes it difficult to determine the average usage per vehicle, which 
is needed to determine the cost per unit usage. Furthermore the units for the 
OPTEMPO were inconsistent and in many cases not documented. Of the 88 
organizations listed in line items for regular (non-deployed) vehicles, 77 of them 
were reported in miles per year, five were reported in hours per year, and six 
were left blank. For the deployed units, 35 were reported in miles per year, one 
was reported in hours per year, and 52 were left blank. The 52 that did not report 
could be attributed to that particular vehicle style not having any deployed 
vehicles for that particular year. Based on these numbers and various 
conversations with Marines who have dealt with HMMWV’s in the past, it is 
believed that the correct unit is in miles per year. 


As with the Army, the Marines’ inventories or counts were another 
significant problem area. Of the 88 line items gathered from VAMOSC, only 73% 


included an inventory; however, all 88 included at least one of the five different 
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cost fields included in the data. A graph of the inventories is included in Figure 7, 
and while the drastic drops seen in the Army data are not as evident here, there 
are several oddities that raise concerns. For instance: for the D1125 model for 
years 2005 to 2007, the inventory varied from 669 to 1175 to 854. A similar 
problem occurred for the D0187 model during the same years; the inventory 
varied from 17 to 357 to 74. For FY-2009, no inventories were reported for any 
model. In summary, the USMC data was not analyzed for this thesis due to the 


lack of vehicle level data. 
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The final observation from this data was the Cost Per Unit (CPU). The 
most interesting observation in this area was with the D0187 model. Over the 
course of 6 years, the CPU data varied drastically and was more than 5—7 times 
higher than the other models’ average in 4 out of 6 years. A graph of the CPU is 
shown Figure 8, and the anomaly that occurred with D0O187 is clear to see. 
There is also a trend beginning in 2005 that shows an increase in CPU. 
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USMC HMMWV Cost Per Unit for 1998 to 2008 
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C. SUMMARY 


Data obtained from the Army and Marine Corps was full of anomalies. 
From unexplained drops in inventory to differences in Cost Per Unit of over 
$25,000, the data contained many inconsistencies. While there is plenty of data 
available on weapons systems from the various databases across the services, it 
appears there needs to be some quality control implemented at the entry level. 
For maintenance issues, jobs are typically entered into the system by personnel 
of rank E-5 and below and then reviewed by Non-Commissioned Officers and 
again by Officers. Unfortunately, this review process is not correcting the 
multitude of errors and needs to be revised across the services. Perhaps with 
new technology available, the multiple entry points could have drop-down menus 
and the ability to warn the user of a faulty input. Ultimately, for DoD to conduct 
thorough and, more importantly, conclusive studies, we recommend that the 
issues addressed above regarding the data be corrected. 
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IV. ANALYSIS 


The analysis focused on two of the six metrics used to measure PBL’s: 
Cost Per Unit Usage (CPUU) and Operational Availability. We examined the 
Cost Per Unit Usage over time to determine if any cost savings could be 
attributed to the ILP. Operational Availability was studied to find any trends 
All the 


analysis was conducted in R and Excel 2007 using the data gathered from the 


before and after the inception of the partnership with AM General. 


Army LIDB. Due to the large number of missing data points, some of the data 
had to be repaired and those repairs are documented in Appendix A. 


Two different models from the Army were studied, the M1097 and M998. 
Over 45,000 vehicles were used for this study, 33,761 M998s and 14,832 
M1097s. The data covered the years 2004 to 2009. After repairs and deletions, 
we had nearly 128,000 work orders for M998’s and 41,000 work orders for the 
M1097’s. 


A. COST PER UNIT USAGE 


The cost per unit usage was difficult to study because the usage data was 
missing multiple values and had countless anomalies. After repairs to the data, 
there was still variability. The mean usage per year per was 914 miles for M1097 
and 1,242 for the M998. A table of the number of vehicles having a certain level 
of usage for a year is shown in Table 2. There were over 28,000 times when a 
vehicle recorded zero usage for the year. The average vehicle usage decreased 
over the period studied. A plot of both models’ average yearly usage is shown in 



































Figure 9. 
Miles of 0 1-1,000 1,001- 5,001- >10,000 
Usage 5,000 10,000 
M998 20,547 26,640 21,362 2,515 851 
M1097 8,001 11,999 5,673 570 254 
Table 2: Number of Vehicles with Established Mileage Per Year 
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Figure 9: Mean Usage per Year 


A boxplot of the log of the usage is provided in Figure 10 and Figure 11 for 
the M1097 and M998, respectively. We chose the log of the usage value 
because the spread of the values was so large it would have been difficult to 
display otherwise. Note that in these plots values of zero were excluded. This 
eliminated from between 9% to 42% of the data for any given year. The 
percentage of zero values is shown below the year in parentheses. Notice that 
the percentages increase over time. These plots indicate a downward drift in the 
usage of the period studied. 
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Boxplot of Log of M998 Usage 








att a ee a we 
en oe 

| 58868 

oO) | 

S ! ! ! : | . 

5 | ! ! | ! ! 

oe : ! ! ! ! ! 

2 ! ! ! } 

g ! ! 
| ] 
ro) ] 

g 5 i 8 
O 
yf F 6 
7 
(15%) (18% (27%) (38% (40% (42% 
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zero from original data 
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Boxplot of Log of 1097 Usage 
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Figure 11: Boxplot of Log of Usage for M1097 excluding all values 
of zero from original data 


Despite the problems with the usage data, we examined the CPUU for 
both vehicle models. The goal was to use the years prior to the implementation 
of the ILP as a baseline and then observe how the metric changed with respect 


to the baseline after implementation. 
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All CPUU values discussed in this chapter account for inflation and are 
displayed in Fiscal Year 2011 dollars. The adjustments were made in 
accordance with the Naval Center for Cost Analysis using the Operations and 


Maintenance Army Appropriations index. (Naval Center for Cost Analysis, 2010) 


1. Summary Statistics 


Table 3 summarizes the CPUU before and after the ILP was implemented. 
The year 2006 is excluded in column three to eliminate the possibility of the 
CPUU being skewed in that year due to initial start-up costs for the program. 
However, notice that when 2006 is included, the CPUU was actually lower. We 
conjecture that when the program initially started, there were still parts on the 
shelves at the depots that had been procured through the old system, and the 
technicians were using these despite the fact that AM General was under 
contract to provide the parts. The key takeaway is that the CPUU in 2006 cannot 
be accurately represented and should be viewed cautiously when trying to draw 


conclusions about the ILP’s success (or lack thereof). 























2004-2005 2006-2009 2007-2009 
M998 $98.14 $111.19 $119.02 
M998 5% Trim $5.72 $10.44 $11.63 
M1097 $80.32 $107.56 $111.75 
M1097 5% Trim $6.21 $8.39 $8.81 














Table 3: CPUU Before and After ILP Implementation 


Based on these averages, the overall CPUU for both units appears to be rising 
over time. We used the 5% trimmed means because there were a small 
percentage of very large values that had significant influence on the CPUU 


values. The distribution of the CPUU values is shown in Table 4. 
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CPUU FY11S$ 
Model & 
Veni $0.00- $5.01- $10.01- | $50.01- | $100.01- 
$5.00 $10.00 $50.00 $100.00 | $1000.00 | >$1000.00 | NA's 
mg998 Count |} 33633 4864 6502 1402 2224 850 20547 
% 68% 10% 13% 3% 4% 2% 
M998 Count 15894 2006 2438 481 792 356 4643 
2004-2005 % 72% 9% 11% 2% 4% 2% 
M998 Count} 17739 2858 4064 921 1432 494 15904 
2006-2009 % 64% 10% 15% 3% 5% 2% 
M1097 Count 10441 2033 3183 885 1190 369 8001 
% 58% 11% 18% 5% 7% 2% 
M1097 Count 1006 127 234 64 58 20 225 
2004-2005 % 67% 8% 16% 4% 4% 1% 
M1097 Count 9435 1906 2949 821 1132 349 7776 
2006-2009 % 57% 11% 18% 5% 7% 2% 
Table 4: CPUU Counts and Percentages for Cost Ranges 
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Figure 12 displays the 5% trimmed average CPUU for both vehicle models for 
each quarter over the period studied. Again, it is clear that the CPUU appears to 


be rising with time. 


5% Trimmed Mean CPUU by Quarter 
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Figure 12: 5% Trimmed Mean CPUU by Quarter from FY2004 to 
FY2009 


Notice that there are several large spikes in the CPUU, especially for the M1097 
vehicle. In the second quarter of fiscal year 2008, there was a large jump for 
both models. This graph displays how inconsistent this metric is and we believe 


that is attributable to variability in the usage data. 


While the quarterly data was not very conclusive, we were able to see a 
better trend when using the yearly averages. Figure 13 and Figure 14 display 
the yearly averages with a trim level of 2%, 5%, and 10%. Notice that in 2006 

ar 


the CPUU increased for both vehicles with the exception of the 2% trimmed 
mean for the M998 vehicle. As previously mentioned, this was expected as a 
response to increased start-up costs. The following year, 2007, the CPUU 
dropped for the M1097, indicating that the program may have been achieving 
some cost savings. However, the following two years were higher and indicate 
that the CPUU was growing with time. We need to point out that over time the 
average usage for these vehicles decreased, as shown in Figure 9. So, for 
routine maintenance that is based on time, not mileage, the CPUU would be 
driven up. This could explain why an increase in the CPUU is seen during the 
period of study. 


M998 Trimmed Mean CPUU 








CPUU FY11$ 
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Years 


Figure 13: M998 Trimmed Mean CPUU from 2004-2009 
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M1097 Trimmed Mean CPUU 
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Figure 14: M1097 Trimmed Mean CPUU from 2004-2009 


Additionally, we looked at the boxplot of each year’s CPUU values side by 
side. Looking at the raw data, it was hard to gather any useful information 
because there was such a large number of outliers and the ranges of values 
were from $0.00 to nearly $300,000. To make this a more readable plot, we 
plotted the log of the CPUU. The plots for both models are shown in Figure 15 
and Figure 16. Again, notice the upward drift over the period studied. While this 
does not indicate any huge shift in costs, it clearly shows that the costs are rising 
over time, and that there is no cost savings that can be attributed to the ILP. This 


may have happened regardless of the ILP, but there is no way to determine that. 
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M998 Distributions by Year 
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Year-by-Year boxplot of Log(CPUU) for M998 


Figure 15: 
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1097 Distributions by Year 
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Figure 16: Year-by-Year boxplot of Log(CPUU) for M1097 


2. CPUU by Year of Manufacture 


Next, we looked at the CPUU based on the Year of Manufacture (YOM). 

Prior to examining this, we expected to see some trends in the data because as 

a vehicle gets older we expect it to need repair more often than a newer vehicle. 

This is not, however, what we observed from the data. The later model vehicles 

had more variability and their CPUU was higher. This may be attributed to newer 
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parts being more expensive, the price of labor rising, or new technology. Plots of 
the two models’ CPUU based on YOM are displayed in Figure 17 and Figure 18. 
Notice that the pre-ILP CPUU values are lower for most years. Although these 
are 10% trimmed means, large spikes in the CPUU values are still visible. The 
one value that is not displayed on the M1097 chart was over $280 per mile. The 
values after the implementation of the ILP have less variability but are generally 
higher than those values prior to the ILP. 


M998 CPUU by Year of Manufacture 
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Figure 17: M998 CPUU by Year of Manufacture before and 
after ILP Implementation 
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M1097 CPUU by Year of Manufacture 
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Figure 18: M1097 CPUU by Year of Manufacture before and 
after ILP Implementation 


3. CPUU by Year of MACOM 


Next, we looked at the CPUU for each Major Army Command (MACOM). 
Two of the MACOMs, Forces Command (FORSCOM) and National Guard 
Bureau (NGB), owned 70% of the vehicles studied. The remaining 30% of 
vehicles were distributed among 15 other MACOMs. Again, we attempted to 
compare the CPUU before and after the implementation of the ILP. The results 
were the same as in the previous analyses. The CPUU for 2004 and 2005 was 
cheaper than for 2006 to 2009. For the FORSCOM and NGB MACOM’s, the 
CPUU was 1.5 to 2.25 times larger after the implementation of the ILP than it 
was before. Again, we used a 10% trimmed mean to eliminate the influence of 
some of the unusually large values. A summary of our findings is shown in Table 
5. A score of one was assigned for those MACOM’s in which the pre-ILP CPUU 
was lower than the post-ILP. A zero was scored for the opposite. The 


43 


percentages at the bottom of the “Score” columns represent the proportion of 
vehicles that belonged to a MACOM in which the costs went up. This indicates 
that the CPUU has risen since the inception of the ILP. 
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M998 M1097 
MACOM Sample 7 sacle - 

Size <=2005 >2005 Score Size <=2005 >2005 Score 
EUSA 1471 2.0% S 241 § 13.20 1 201 0.8% § 1.75 S 34.13 1 
Lrorscom | 306aa_| 426% | 5 soa | $s aaa | 1 | | 1658 | crsm | ¢ sos | 5 1m | 1 | 
IMCOM 446 0.6% S 3.65 S$ 4.06 1 672 2.5% § 1.50 S 24.06 1 
MDW 16 0.0% S$ 9.72 S$ 1.57 0 1 0.0% NA S$ 0.02 NA 
NETCOM 417 0.6% S 4.23 § 10.61 1 188 0.7% § 12.21 S 36.85 1 
[nes | ts2a7_ | 242% | 5 a5 | s 273 | 1 | | 27 
TRADOC 3281 4.6% § 1.02 S$ 2.44 1 327 1.2% S$ 2.26 S$ 2.69 1 
USACIDC 34 0.0% S 42.88 $ 280.27 1 0 0.0% NA NA NA 
USAIMA 34 0.0% § 10.97 S$ 9.23 0 0 0.0% NA NA NA 
USAMC 53 0.1% S 2.81 S 36.31 1 1414 5.3% NA S 29.36 NA 
USARC 11943 16.6% S$ 3.74 S 4.66 1 2251 8.5% S 462 S$ 3.87 0 
USARCENT 179 0.2% § 241 S 22.64 1 0 0.0% NA NA NA 
USAREUR 7A 4917 6.8% S 2.67 S$ 5.63 1 440 1.7% S$ 4.42 S$ 5.65 1 
USARPAC 2286 3.2% S 5.43 S$ 7.83 1 1117 4.2% § 2.05 S 23.09 1 
USARSO 6A 114 0.2% S 0.01 S$ 0.99 1 0 0.0% NA NA NA 
USASOC 534 0.7% S$ 2.64 § 7.15 1 541 2.0% § 12.03 S$ 5.16 0 
USINSCOM 295 0.4% § 10.71 S 9.86 0 19 0.1% § 5.37 S$ 6.23 1 
TOTAL 71890 99.52% 26497 84% 





Table 5: 


CPUU Comparison by MACOM 
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These analyses revealed that the CPUU has risen over the six-year period 
studied. Additionally, comparing the baseline years, 2004—2005, to the post-ILP 
years, 2006-2009, we noticed the baseline years’ CPUU values are consistently 


lower. 


4. Statistical Testing 


It is important that we point out that for this study we had the entire 
population of Army HMMWV’s for which data was available. Therefore, strictly 
speaking, there is no call for the testing of statistical hypotheses, in which data 
from a sample is used to draw inferences about parameters of a population. 
However, it may be reasonable to consider the set of yearly HMMWV 
observations as if they were a sample from a larger, hypothetical population of 
possible observations. In this way, the observed data can serve as a sample 
from this larger population, which may represent future observations on these (or 
other) vehicles, and hypotheses about this population (and in particular whether it 


has changed) can be tested. 


In order to have a more robust test, we looked at only those vehicles 
containing values for the CPUU in both time periods. Specifically, we looked at 
two-year comparisons. There were eight different combinations of two-year 
comparisons. Vehicles that had zero usage for a given year but had associated 
costs for that same year, had a N/A value for CPUU. We eliminated those 
vehicles that had a N/A for the CPUU value in either year. We then conducted 
the t-test on the difference of the log of the CPUU for both time periods. The log 
was used because the assumptions of Normality required for the t-test was not 
plausible for the original data. However, after transforming the data using the 
logarithmic function, the data appears Normal for both models. This is displayed 
in Figure 19 for the M998 vehicle. While not every year combination for each 
vehicle is shown, each of the plots are similar to those in Figure 19. 
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Figure 19: Histogram of M998 Log of CPUU Difference (2004 vs. 


Post-ILP Years) 
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We also conducted the Wilcoxon test. For this test, the null hypothesis is 
the same as the t-test: the average change in CPUU between the two years 
being compared is equal to zero. We conclude from this test that the difference 
in means is different and that the pre-ILP years are cheaper. Additionally, we 
conducted the Chi-Squared test to ensure consistency in our results. For the 
Chi-Squared test, the null hypothesis is that the percentage for which the 
difference between CPUU values is less than zero is equal to 50% (there are as 
many vehicles that had a higher CPUU Pre-ILP as there were that had a higher 
CPUU Post-ILP). The final column in Table 6 and Table 7, True-p, is the 
percentage of vehicles that had a higher Pre-ILP CPUU than the Post-ILP CPUU. 
For both models in nearly every instance, this value is less than 50%, which 
indicates that in most cases, the CPUU was lower prior to implementation of the 
ILP. These results confirm that on average the Pre-ILP CPUU is lower than the 
Post-ILP CPUU. Table 6 and Table 7 contain the results of all three tests for 
both models. 
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T-Test Wilcoxon Chi-Squared 











Ha = Difference in means is NOT zero Ha = Difference in means is NOT zero Ha = True pis not equal to 0.5 
Log(Old-New) Old-New Old>New 

T-stat Df P-value | Result V P-Value | Result Chi*2 P-value | True-p 
04 vs 06 2.20E-16 1139967 2.20E-16 8.28E-16 
04 vs 07 2.20E-16 414444 2.20E-16 2.20E-16 
04 vs 08 2.20E-16 307282 2.20E-16 2.20E-16 
04 vs 09 2.20E-16 168809 2.20E-16 2.20E-16 
05 vs 06 5.84E-01 4883129 2.92E-01 1.57E-01 
05 vs 07 1.91E-01 2112927 4.09E-04 8.31E-04 
05 vs 08 2.20E-16 1354358 2.20E-16 2.20E-16 


05 vs 09 1.70E-13 515748 2.50E-12 1.99E-12 





Table 6: M998 Two-Year Comparison Test Results 











T-Test Wilcoxon Chi-Squared 
Ha = Difference in means is NOT zero Ha = Difference in means is NOT zero Ha = True p is not equal to 0.5 
Log(Old-New) Old-New Old>New 
T-stat Df P-value | Result V P-Value | Result Chi42 P-value | True-p 
04 vs 06 3.80E-12 8.50E-14 1.29E-10 
04 vs 07 8.22E-11 2.68E-11 4.34E-10 
04 vs 08 4.02E-11 5.05E-10 1.34E-06 


04 vs 09 1.35E-06 6.51E-06 1.98E-04 
05 vs 06 8.30E-01 3.50E-01 1.00E+00 
05 vs 07 6.20E-02 8.80E-03 4.57E-03 
05 vs 08 3.43E-05 1.84E-05 1.14E-03 
05 vs 09 7.93E-04 2.30E-04 6.25E-04 





Table 7: M1097 Two-Year Comparison Test Results 
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5. Mixed Effects Model 


The final piece of our analysis involved using a mixed effects ANOVA 
model. (Galwey, 2006) In this model, the response is CPUU. The effect of year 
is “fixed” in that the levels of year are known in advance, but the effect of vehicle 
is “random” in that we envision (at least hypothetically) the observed vehicles 
acting as a random sample from a larger population of possible vehicles. This 
approach allows us to consider every observation for every vehicle, instead of 
simply comparing years two at a time, while adjusting for variability among 
vehicles. As with the statistical testing, we used the log of the CPUU because 
the assumption that the errors are Normal was not plausible for the original data, 


but for the log, this assumption was valid. 
Specifically, the model proposes that 
Cy=uty+ Vit Ej 
where 
u = intercept, 
y; = jth year effect (fixed), 
V, = ith vehicle effect (random, assumed N(0, 0’), and 
€j = error (assumed iid N(O, o-°)) 


The null hypothesis of interest is that the year effects are identically zero (that is, 
that overall average CPUU has remained constant across years). This model 
was implemented using the Ime4 package in R. (Bates, 2010) That package 
does not directly provide p-values for the F test associated with that null 
hypothesis, but it is possible to compare two nested models. We compared the 
above model to a similar model that included vehicle but excluded year. The 
resulting p-value was smaller than .01, leading us to reject the null hypothesis of 
constant average CPUU and to conclude that the average CPUU value has 
changed with time. The same conclusion is reached when comparing the two 
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models’ AIC values. Since the model that included the year had a smaller (that is, 
better) AIC than the one without, we reject the null hypothesis. 


To examine the effect of the ILP more directly, we built a third model in 
which the year was represented by a two-level factor denoting the pre-ILP (2004- 
2005) and post-ILP (2006-2009) years. We can then examine the null hypothesis 
of no effect of the ILP. In the comparison of this model to the baseline (vehicle- 
only) model, we observe a slightly smaller AIC, and the p-value associated with 
that null hypothesis was less than .01, suggesting that the imposition of the ILP 
had a significant effect on average CPUUs. Based on our previous analysis and 
this test, it is safe to conclude that the ILP is not providing any cost savings to the 
DoD. 


B. OPERATIONAL AVAILABILITY 


The availability data collected from LIDB was not broken down by vehicle 
like the cost and usage data. Instead, it was broken down by Unit Identification 
Number (UIC), Location, and MACOM. Each UIC reported a Mission Capable 
(MC) percentage on the 15th of every month. This is the metric we used for this 
study. 


First, we looked at the Operational Availability versus the calendar quarter. 
This showed a fairly constant rate until the latter part of 2007 when there were 
noticeable reductions in the Operational Availability rate. For the M998 vehicle, 
we see a Slight increase in readiness until that point in 2007 when the rate drops 
about 2%. The M1097 falls 10% between the 2nd quarter of FY2007 and the 
2nd quarter of FY2008. They both appear to level out near 89-90% by the end of 
2009. It is clear that something occurred in 2007 to cause a drop in readiness. It 
is also clear that the Operational Availability rates prior to the ILP implementation 
were higher than those after. These results are displayed in Figure 20. The 
surge in Iraq occurred in 2007, and this could have had some effect on this 


metric. Also, as documented in Chapter Ill, the number of M1097 vehicles 


51 


increased between 2006 and 2009. An increase in vehicles without an increase 
in logistical support could have driven the average Operational Availability down. 


Mean Availability by Quarter 


Mission Capable Percentage 
0.90 0.95 1.00 


0.85 
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Figure 20: Mean Operational Availability of Vehicles by Quarter 


As in the CPUU analysis, we looked at the Operational Availability by 
MACOM before and after the implementation of the ILP. The results were not as 
conclusive as with the CPUU. While the post-ILP Operational Availability 
numbers were on average lower than those from before, Operational Availability 
for the two MACOM’s that own the majority of the vehicles remained fairly 
constant. In the case of the M998 at FORSCOM there was a 3% increase, 
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indicating an improvement since the ILP implementation. But, Army Materiel 
Command (USAMC) had large drops in Operational Availability for both vehicle 
models. Considering that AMC owned less the 10% of the M1097s and less than 
1% of the M998s, this large drop does not lead us to any conclusions. A 


summary of the results are presented in Table 8. 



































































































































MACOM M998 M1097 
<=2005 >2005 | Score <=2005 >2005 | Score 
EUSA 92.84% | 94.96% 1 94.54% | 92.68% 0 
IMCOM NA 94.35% NA NA 99.86% NA 
MDW 93.67% | 99.31% 1 98.41% | 99.50% 1 
NETCOM 94.60% | 94.32% 0 97.67% | 96.50% 0 
[New | 89.90% 90.77% | 1 | | 95.54% | 95.43% | 0 | 
OSD 97.82% | 97.21% 0 NA 100.00% | NA 
USACIDC 94.65% | 96.10% 1 100.00% | 99.59% 0 
USAIMA 84.25% | 100.00% 1 97.05% | 99.99% 1 
USAMC 88.95% | 65.31% 0 83.24% | 69.09% 0 
USARC NA 90.43% NA NA 95.47% NA 
USARCENT NA 95.72% NA NA 94.03% NA 
USAREUR 
7A 92.13% | 89.52% 0 94.34% | 97.17% 1 
USARPAC 92.43% | 93.18% 95.01% | 90.69% 0 
USARSO 
6A NA 93.25% NA NA 97.08% NA 
USASOC 96.67% | 97.48% 1 98.28% | 99.08% 1 
USINSCOM | 93.62% | 94.61% 1 94.30% | 95.85% 1 
Averages 92.39% | 92.88% 95.30% | 95.21% 
Final Score Percentage 0.67 0.55 
Table 8: Operational Availability Comparison by MACOM 


Histograms for the Operational Availability of the M998 vehicle before and 
after implementation of the ILP are displayed in Figure 21. Plots of the M1097, 
while not shown here, displayed similar results. Based on these plots and the 
results from above across MACOMs, the Operational Availability does not appear 
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to change in any significant manner. In fact, it appears to hold relatively constant 
between the 90-95% range. This has been the DoD unwritten standard for 


weapons systems Operational Availability. 


Histogram of Pre-ILP Availability for M998 
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Figure 21: Comparison Histogram of Operational Availability for 
M998 Vehicle 
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C. CONCLUSION 


This study analyzed two of the DoD-defined metrics to determine if the 
HMMWV ILP was functioning better than the traditional maintenance plan. 
Several different approaches were taken to view the performance of the program 
using the CPUU and Operational Availability. There appeared to be an increase 
in cost and no significant change in Operational Availability, over the six-year 
period studied. Hypothesis testing was conducted on the metrics for the two time 
periods, and it revealed that there was, indeed, a statistically significant 
difference between the two time periods for CPUU. To further confirm this, a 
year-to-year comparison was conducted as well and the results were confirmed. 
A mixed effects model was built, and it was determined that the year had a 
significant relationship with the CPUU. On average, the CPUU prior to 
implementation of the ILP is lower than the CPUU after; however, it cannot be 
determined that the ILP is the cause of this decrease. The only conclusive 
statement that can be made is that the ILP has not provided the DoD with cost 


savings. 


The accuracy of the data was an area of concern for the writer and much 
time and effort was exhausted to correct it. However, there is still doubt as to its 
complete validity. Since the metric studied was based on the usage data, there 


could be inherent errors in these results. 
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V. SUMMARY AND RECOMMENDATIONS 


The Department of Defense has claimed that Performance-Based 
Logistics is the preferred method of Life-Cycle Support for weapons systems. 
This approach was implemented in an effort to increase availability while 
decreasing costs. Unfortunately, based on the results found in this study, neither 


of these results appears to have occurred in the case of the Army HMMWV ILP. 
Four principal conclusions were found in the course of this study: 
1. CPUU had an upward trend from 2004 to 2009; 


2. CPUU was lower before the ILP and higher after the ILP 


implementation; 


3: Operational Availability showed no significant trend from 2004 to 
2009; and 

4. Operational Availability experienced no significant change due to 
the ILP. 


The CPUU rose continuously over the time period studied. While there 
were some drops in certain quarters, the overall trend was upward. While 
unadjusted costs are expected to grow over time, this study accounted for 
inflation and still a rise in costs was observed. Additionally, when the two time 
periods were compared, the time period before the ILP was determined to have a 
smaller CPUU than the post-ILP time period. This was found through several 
different analyses and confirmed using statistical hypothesis testing. Again, we 
point out that the CPUU is driven by the usage data, which we found to be a 
major area of concern in this study. However, every effort was made to use that 
data that was available and correct obvious errors in a logical manner. The 
CPUU rise was subtle and could be attributed to rise in costs over time. It could 
also be attributed to the fact that usage went down over this same time period. If 
there are time-related maintenance requirements, then the CPUU would be 
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driven up because the maintenance is still conducted regardless of the usage. 
The costs of the HMMWV program could have seen a similar (or even greater) 
rise even if the ILP had not been in place. 


While the CPUU rose over the study period, the Operational Availability 
did not show any change. While there appears to be a slight drop in the latter 
portion of 2007, this decrease was minute and the overall Operational Availability 
remained above 90% throughout the period studied. The slight drop could have 
been attributed to other things such as, perhaps, the surge in Iraq in 2007. This 
metric remained relatively constant for both models studied until that time frame, 
at which point a decrease was seen. Even after the decrease, both models were 
maintaining nearly 90% Operational Availability, which is the unwritten minimum 
throughout the DoD. While the DoD wants to achieve cost savings, as well as 
availability increases, the availability metric is of higher importance. Operational 
Availability is the key for our warfighters to be able to conduct their missions 
successfully. Regardless of costs, if weapons systems are not available for use, 


we will have many things to be concerned about other than costs. 


While these conclusions are important, the primary finding in this study 
was that the data to conduct an analysis of PBLs is simply not accurate enough. 
In order to give decisions makers a tool to plan for future Life-Cycle Sustainment 
of weapons systems, analysts need sufficient and accurate data that can be used 
to analyze the DoD prescribed metrics. The key problem encountered in this 
study was the usage data obtained from the Army LIDB database. Missing 
values and obvious errors were rampant in the data. While we went to great 


efforts to correct these errors, this could still have an effect on the outcome. 


The Army LIDB is an improvement from the past and has the capability to 
provide all the necessary data for analysts to conduct studies such as this. 
However, the inputs into this database are clearly in need of refinement. Most 
likely, these errors are being committed at the lowest level, which is when the 
maintenance man goes out to take the monthly reading and enters it into the 


computer. There needs to be more quality control on this process. The best 
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solution would be for the Army to develop electronic tracking of the data, and 
have a system similar to the Navy’s remote monitoring system being installed on 
ships. This system allows the watchman to simply plug in a hand-held device to 
a piece of equipment and the operating parameters are automatically uploaded. 
The Army could develop a similar device such that each time it is plugged in to a 
vehicle, the current odometer reading is uploaded, thus preventing any human 


error. 


The original plan for this study was to use the LSD-41/49 Class diesel 
engines, as well as the HMMWYV, to examine the effect of ILP. The Navy made a 
shift in diesel maintenance practices on the LSD-41/49 Class ship in the past 
decade after one ship failed to deploy and three other ships deployed but were 
essentially ineffective. (Caccese, 2007, p. 4) The basic idea behind this new 
practice, Diesel Readiness System, was to increase Operational Availability and 
reduce costs. These goals would be accomplished through a one-time base line 
assessment, followed by more proactive (vice reactive) maintenance schedules. 
Some of the major changes would include automated collection of machinery 
history logs, standardized repair requirements, new failure and costs metrics, off- 
ship lube oil analysis, and program management for business rules. (Caccese, 
2007, p. 6) 


While the DRS program has made improvements over the previous diesel 
maintenance program, it would be interesting to see if a Performance-Based 
Logistics (PBL) contract with the Original Equipment Manufacturer (OEM) would 
produce even more costs savings for the Navy. Fairbanks Morris Corporation 
(FMC) and Northrop Grumman have offered to establish such a partnership for 
the LSD class ship. The offer has never come to fruition; however, FMC was 
contacted and asked about a possible maintenance plan for these engines. FMC 
was kind enough to release some figures on what they would have charged the 
Navy for a 30-year maintenance program, which would require the engines to be 


maintained at 90% Operational Availability or better. 
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One of the key topics of concern amongst Navy brass is Total Ownership 
Costs (TOC) for diesel engines. In our study, the TOC was going to be the 
metric of interest for the LSD 41/49 Class ship. Unfortunately, there was not 
enough data or historical information to accurately calculate the TOC in the time 
allotted. Therefore, the diesel engine was abandoned and the study was 
narrowed down to the Army HMMWV only. 


While the Navy is developing monitoring systems, as mentioned 
previously, its database is years behind that of the Army’s. The Navy and Marine 
Corp Visibility and Management of Operating and Support Costs (VAMOSC) 
database contains a myriad of data, but gives summary statistics instead of 
weapon system level data. It is important for the analysts to have data that is at 
the weapon system level. Furthermore, VAMOSC is difficult to use. The queries 
are not intuitive and have too many options that leave the user unsure about the 
type of report he or she is producing. Additionally, it is difficult to export the data 
into a Comma Separated Value (CSV) or some other format that is commonly 
used by analysts. So, the Army and Navy need to collaborate and share their 
strengths when it comes to data collection and storage. This would provide 
analysts with accurate data that in turn could provide decision makers sound 
decision-making tools for future planning of weapons systems life-cycle 
sustainment. That the data was dirty and missing came as no surprise, as many 


of the studies reviewed had similar findings, as documented in Chapter II. 


There are ample opportunities for future studies on Performance Based 
Logistics. The DoD has prescribed five metrics: 1) Operational Availability, 2) 
Operational Reliability, 3) Cost per Unit Usage, 4) Logistics Footprint, and 5) 
Logistics Response Time. This thesis addressed Operational Availability and 
CPUU, but follow on studies could address the other metrics. Furthermore, other 
platforms under PBL contracts could be studied using the same metrics, any 
combination of the others, or all five of them. 


60 


APPENDIX: DATA CORRECTION 


This appendix discusses various refinements we made to improve the 


quality of the data for our analysis. 


A. SERIAL NUMBERS 
1. WON Data 


The serial number data for several WONs from LIDB required some 
correction. Based on the RAND study, the data from LIDB, and emails from 
Army Materiel Command, it was determined that a valid serial number was a 
three- to six-digit number (Neal, 2010; RAND Corporation, 2008, 56). Originally, 
this data contained 88,036 unique serial numbers from 264,554 WONs. In our 
data, 2,092 serial numbers were entered along with their corresponding 
registration number separated by either a slash or a dash. The registration 
number was removed from those entries. There were 582 entries that contained 
only the registration number. We replaced this with the corresponding serial 
number for each vehicle. There were 2,116 entries which had a valid six-digit 
number either preceded or followed by another character. These characters 
were removed leaving only the six-digit number. We removed 20,513 entries 
because the serial number entry was unable to be repaired or it had not been 
entered. After these repairs and eliminations, 72,394 unique serial numbers 


remained for analysis. 


2. Usage Data 


The serial number entries in the usage data gathered from LIDB was in 
great condition and only required a few eliminations. For the M998 model, the 
original data had 566,240 line items from the years 2000 to 2009. From these 
line items we found 48,266 unique serial numbers. We eliminated 204 of these 
entries due to registration numbers not corresponding to serial numbers. For 


example, the registration number was recorded as being the same for multiple 
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serial numbers. We were left with 48,062 serial numbers for the M998 model. 
For the M1097 model, the exact same process for elimination was used resulting 
in an original unique set of serial numbers of 25,349 being reduced to 25,055. 


3. Combining the Serial Number Data 


Using the serial number lists created from each of the two data sets, we 
created a master list. The two models combined had 72,394 serial numbers from 
the WON data and 73,117 from the usage data. A total of 52,147 numbers 
appeared in both lists. Of that, 35,590 were M998s and 16,584 were M1097s. 
This resulted in 142,186 WONs for M998 and 46,430 WONs for M1097. From 
the original WON data, we eliminated 29% of the line items from the repair, 


elimination, and matching of the serial numbers across both data sets. 


B. USAGE DATA 


The vehicle odometer data gathered from LIDB had a large number of 
missing values and other errors, such as double entries that did not correspond, 
and unusually high readings from one month to the next. First, any values that 
were over 300,000 were removed. The data was then cut into 50,000 mile 
categories across all the years, 2000-2009, for each vehicle. Each reading was 
assigned a category between one and six (0-50,000=1, 50,001—100,000=2, etc.). 
Next, the mode of these categories for each individual vehicle was determined. 
Values that fell in the mode of the categories plus or minus one increment were 
kept. For example, if a vehicle had a mode of three, then all values that were 
assigned a category of two, three, or four were retained. All others were marked 
as “N/A.” This same process was repeated for each year with the cuts being 
reduced to 8,000. The process was repeated once more for each quarter using a 
3,000 mile cut. This process eliminated the cases where we had anomalies in 
the month-to-month readings. In cases where there were multiple entries for one 
vehicle for a single year, the entries were merged. If one entry had a value fora 
particular month and the other entry did not, then the value was retained. If there 
were multiple different entries for a single month, then the maximum entry was 
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retained. Next, we looked through each reading for a single vehicle and if the 
previous months reading was higher than the current months reading, the current 
months reading was replaced with “N/A.” We calculated the actual usage by 
subtracting the n+1 month from each month assuming that the readings were 
taken on the final day of each month. Because of the eliminations, we used 
simple imputation to fill in gaps in the data. Then, the sum of the monthly data 


was calculated for each vehicle for each year. 


1. Cost 


The cost data was associated with each individual WON, and therefore, 
needed to be summed in order to determine total costs for a vehicle for each 
year studied. Using the remaining WON data after the serial number corrections, 
we built a new database that contained the sum of the costs for each vehicle for 
each year. We also counted the number of WONs for a particular vehicle for 
each year and included that information in the new database. The primary metric 
studied, CPUU, was calculated by dividing the total cost for a vehicle for the year 
by the total usage for that year. These values were then adjusted for inflation 
and converted to FY11$. The Operations and Maintenance Army Appropriations 
index was used for this adjustment. (Naval Center for Cost Analysis, 2010) 
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